{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![LOGO](../../../img/MODIN_ver2_hrz.png)\n",
    "\n",
    "<center><h2>Scale your pandas workflows by changing one line of code</h2>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 2: Speed improvements\n",
    "\n",
    "**GOAL**: Learn about common functionality that Modin speeds up by using all of your machine's cores."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concept for Exercise: `read_csv` speedups\n",
    "\n",
    "The most commonly used data ingestion method used in pandas is CSV files (link to pandas survey). This concept is designed to give an idea of the kinds of speedups possible, even on a non-distributed filesystem. Modin also supports other file formats for parallel and distributed reads, which can be found in the documentation.\n",
    "\n",
    "We will import both Modin and pandas so that the speedups are evident.\n",
    "\n",
    "**Note: Rerunning the `read_csv` cells many times may result in degraded performance, depending on the memory of the machine**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import modin.pandas as pd\n",
    "import pandas\n",
    "import time\n",
    "import modin.config as cfg\n",
    "cfg.StorageFormat.put(\"hdk\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dataset: 2015 NYC taxi trip data\n",
    "\n",
    "We will be using a version of this data already in S3, originally posted in this blog post: https://matthewrocklin.com/blog/work/2017/01/12/dask-dataframes\n",
    "\n",
    "**Size: ~200MB**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We download data locally because currently `HdkOnNative` doesn't support read files from s3 storage.\n",
    "# Note that this may take a few minutes to download.\n",
    "\n",
    "import urllib.request\n",
    "url_path = \"https://modin-datasets.intel.com/testing/yellow_tripdata_2015-01.csv\"\n",
    "urllib.request.urlretrieve(url_path, \"taxi.csv\")\n",
    "path = \"taxi.csv\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `pandas.read_csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "\n",
    "pandas_df = pandas.read_csv(path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"])\n",
    "\n",
    "end = time.time()\n",
    "pandas_duration = end - start\n",
    "print(\"Time to read with pandas: {} seconds\".format(round(pandas_duration, 3)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Expect pandas to take >3 minutes on EC2, longer locally\n",
    "\n",
    "This is a good time to chat with your neighbor\n",
    "Dicussion topics\n",
    "- Do you work with a large amount of data daily?\n",
    "- How big is your data?\n",
    "- What’s the common use case of your data?\n",
    "- Do you use any big data analytics tools?\n",
    "- Do you use any interactive analytics tool?\n",
    "- What’s are some drawbacks of your current interative analytic tools today?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `modin.pandas.read_csv`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "\n",
    "modin_df = pd.read_csv(path, parse_dates=[\"tpep_pickup_datetime\", \"tpep_dropoff_datetime\"])\n",
    "\n",
    "end = time.time()\n",
    "modin_duration = end - start\n",
    "print(\"Time to read with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
    "\n",
    "print(\"Modin is {}x faster than pandas at `read_csv`!\".format(round(pandas_duration / modin_duration, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Are they equals?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "modin_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concept for exercise: Groupby and aggregate\n",
    "\n",
    "In pandas, you can groupby and aggregate. We will groupby a column in the dataset and use count for our aggregate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "\n",
    "pandas_groupby = pandas_df.groupby(by=\"total_amount\").count()\n",
    "\n",
    "end = time.time()\n",
    "pandas_duration = end - start\n",
    "\n",
    "print(\"Time to groupby with pandas: {} seconds\".format(round(pandas_duration, 3)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = time.time()\n",
    "\n",
    "modin_groupby = modin_df.groupby(by=\"total_amount\").count()\n",
    "\n",
    "end = time.time()\n",
    "modin_duration = end - start\n",
    "print(\"Time to groupby with Modin: {} seconds\".format(round(modin_duration, 3)))\n",
    "\n",
    "print(\"Modin is {}x faster than pandas at `groupby`!\".format(round(pandas_duration / modin_duration, 2)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Are they equal?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pandas_groupby"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "modin_groupby"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Please move on to [Exercise 3](./exercise_3.ipynb)**"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
